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district administrators, and entry-year teachers. The people who are the 
intended focus of the Teacher Evaluation Program (TEP) are referred to as 
interns. Interns are assisted by consulting teachers who are experienced 
teachers assigned to orient, help, and evaluate the interns. Consulting 
teachers receive specific guidelines and must complete reports at given times 
in the TEP process. The case study began with a six-person focus group to 
discuss the TEP program. A quantitative and qualitative questionnaire was 
developed and revised, and then sent to approximately 200 participants. 
Responses from all stakeholders supported the mentor -evaluation program. Some 
stakeholders were more positive than others, but everyone believed the 
program worked. There was a high degree of comfort in having the same 
consulting teacher serve in both mentor and evaluator roles. Participants 
thought that the TEP should not be applied to all participants in the same 
way; experienced teachers newly hired should not receive the same focus as 
novice teachers. Participants also objected to the amount of record keeping 
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EVALUATION OF A TEACHER MENTORING PROGRAM 
USING A MIXED METHODS APPROACH 

Dr. Louis Trenta, Dr. Isadore Newman, Dr. Carole Newman 
The University of Akron 
Dr. James Salzman, Cleveland State University, 

David Lenigan, Berea City School District 
David Newman, The University of Akron 

Objective 

This paper reports on an evaluation of a unique 3-year program designed to support new 
teachers in a school district. The emphasis in the evaluation was on the participation of the 
multiple stakeholders in the evaluation process. The teachers xmion, school district 
administrators, and the entry year teachers jointly supported the program and endorsed this 
evaluation. 

Theoretical Perspective 

When the Joint Committee on Standards for Educational Evaluation considered personnel 
evaluation, it made two observations of interest to school district leadership and, hopefully, the 
school community. First, it noted that it in order to educate students effectively, it is necessary 
for the institution to use evaluation to select, retain, and develop quality personnel (p. 5). 
Secondly, the Committee noted, “However, despite its centrality, educational institutions have 
often been ineffective in carrying out their persoimel evaluation responsibilities” (p. 6). Bolman 
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and Deal reflect somewhat the same point when they write about the substantial time, effort, and 
money devoted to evaluating individuals and programs and note that the results of these 
evaluations typically disappear and yet by their existence serve the purpose of giving the 
institutional an image of good management (p. 244). Yet, they also make the point that 
evaluation, whether of individuals or a program, serves multiple purposes. It can at once serve to 
reward and control performance, help people grow, be an opportunity to exercise power, and 
provide an occasion for all to play roles in a shared ritual (p. 266). 

In the current school reform, high stakes testing environment, the task before the district 
administrator and the evaluator is to overcome the ineffectiveness of evaluation — individual and 
program — and actually cause improvement. This means learning or adaptation must take place; 
the individuals being evaluated must learn and change behaviors and the institution must learn 
and change institutional culture. To bring this about it is necessary for evaluators to “recognize 
the arena in which evaluation operates, understand organizational dynamics, tailor their 
procedures and techniques to market the evaluation process effectively, and become one of the 
voices in the organization” (Wholey, Hatry, & Newcomer, p. 536). Further, Preskill and Torres 
suggest that such individual and institutional (team) learning occurs when individuals participate 
in a collaborative evaluation process that is guided by dialogue and reflection (pp. 26 & 35). 
Heifetz seems to concur when he says. “The inclusion of competing value perspectives may be 
essential to adaptive success” (p. 23). Bridges and Groves in winding up their report on politics 
in personnel evaluation observed that their analysis suggested “that viewing personnel evaluation 
simply in rational, technical terms conceals more than it illuminates” (p. 335). 

With this context in mind, the role of the evaluator must go beyond technical assistance. 
Working with the institutional stakeholders in a chnical maimer, the evaluator and the 
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stakeholders together evolve a highly customized evaluation over time (Preskill & Torres, p. 29). 
Since, and perhaps in part due to, the Qualitative-Quantitative Paradigm Wars of 1980-1995, a 
greater openness to mixed methods in evaluation has occurred (Caracelli, p. 100-101). This 
openness to a variety of methods should facilitate bringing educators of all roles — administrative 
and classroom — together to approach evaluation as a means to promote dialogue, to enhance and 
make more professional satisfying the work of both teachers and administrators (Stronge & 
Tucker, p. 356). This would make all the more important the conceptualization of the role of 
today’s evaluators by the application of “such terms as facilitator, problem solver, educator, 
coach and critical friend’ (Caracelli, p. 103). 

In their research on implementing a new teacher evaluation system, Davis, Pool, and 
Mits-Cash also found. 

Given what is known about change in educational organizations, it is clear that for a 
new teacher assessment system to continue to affect the lives of students and improve 
teaching and learning, continual up-dates, modifications, and revisions are in order. 
Especially as the systems move toward full implementation, it is imperative that 
district policy makers have as much feedback from the field as possible. This 
feedback should be frequent, accurate, and derived from a variety of data collection 
methods [emphasis added] (p. 304). 

Thus, we accepted as a starting point that for an evaluation to be successful the 
participants have to be involved in its process. The stakeholders have to understand and value the 
information that can be derived from the evaluation, and they need to be willing to become 
collaborative partners in the undertaking. To accomplish that we took as a framework for this 
project the Personnel Involvement Evaluation Model (Newman, Deitchman, & Williams, 1978). 
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This model is a four-step process that requires engagement with key stakeholders in all stages. 
The four stages are: 

1 . Identify a sample of key stakeholders 

2. Collaboratively develop the evaluation criteria/instrument 

3. Member check for validity of criteria/instrument 

4. Collaboratively analyze/interpret the data. 

This study began with conversations between the district administrator who was 
responsible for the teacher mentoring-evaluation plan and the researcher. Those discussions and 
discussions with the District Review Board, which is responsible for supervising the 
implementation of the program, brought out the presence of the components of a well-designed 
induction program. Dagenais specifies the five characteristics of successful induction programs 
to be: (a) they provide a clear scope for the program, (b) they identify mentor selection and 
matching criteria, (c) they provide intrinsic and/or extrinsic rewards for attracting and retaining 
mentors, (d) they provide mentor training, and (e) they evaluate the success of the program. The 
district’s induction program had the first four characteristics clearly present. The fifth 
characteristic, evaluation of the program, was considered insufficient to generate program 
improvement and thus the researchers were approached to undertake this study. 

Description of Program 

The object of this evaluation was the a medium sized (8,000 students), mid- west, 
suburban school district’s Teacher Evaluation Program (TEP) as it was implemented for 
beginning teachers, experienced teachers new to the district, and reassigned teachers as well as 
educational persoimel such as librarians and counselors working under the educational persoimel 
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certificate for the first time. The responsibility for implementation and administration of the TEP 
was been assigned to a District Review Board (DRB). 

The people who are the intended focus of the TEP are referred to as interns in the 

I 

program’s documentation and this evaluation. A second role in the TEP is that of consulting 
teacher. Consulting teachers are experienced teachers assigned by the district to orient, help 
develop, and evaluate interns during their first three years of employment with the district. The 
consulting teachers are to provide feedback for development and to evaluate the interns as part of 
the re-employment decision-making process. For each intern there is a designated administrator 
who works with the consulting teacher by discussing the intern’s development plan with the 
consulting teacher and by advising the consulting teacher of concerns related to the intern. The 
designated administrator deals directly with the intern on non-instructional matters such as 
attendance and district personnel procedures and indirectly through the consulting teacher on 
instruction related matters for the first year of intern participation. In the second and third years 
of participation the designated administrator and consulting teacher have equal responsibility for 
evaluation of instructional performance. 

There are at least nine forms specified for use by the consulting teachers and 
administrators to provide official communications with the intern and the DRB. A consulting 
teacher is directed to spend between 20 and 36 hours with his/her intern in classroom 
observations and conferences during the intern’s first year and 13 to 24 hours in the second and 
third years. A very specific timeline of activities and reports is given with specific actions to be 
earned out in September and October, November, between December 1 and February, in 
February, March, and May. 
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Stipends are provided to the consulting teachers to prepare program documents and meet 
with interns, administrators, and the DRB. Class coverage or release time is provided for the 
consulting teacher to do classroom observations and conferences. 

Research Methods 

As noted above, the case study began with conversations between the district 
administrator who was responsible for the teacher mentoring-evaluation plan, the DRB, and the 
researcher. The researcher developed key evaluation concepts from these conversations and used 
those concepts to stimulate discussions with a six person focus group composed of 
representatives from all stakeholder groups in the program (DRB, mentors/consulting teachers, 
and teachers who had recently been mentees/new hires). One of the evaluators took notes 
summarizing the essence of the discussions, checking their accuracy by paraphrasing them to the 
focus group members until the members agreed that the researcher had captured their views. The 
data was compiled and categorized by the evaluators, and the categories were checked with the 
district administrator. 

There were four anticipated results from this stepped process. First, the evaluation 
designers, that is, both the outside evaluators and the district personnel, identified the issues that 
were meaningful and pertinent for the district at this time. Secondly, the evaluators learned about 
the issues from knowledgeable and motivated stakeholders. Thirdly, the stakeholders became 
sensitive to each other’s views about the program and its components. Finally, the evaluators 
were able to gain consensus concerning the constructs and evaluation questions about data 
needed to be collected. The five essential evaluation questions were: 

1 . What is effective in this program as it is currently designed? 

2. What is not effective in this program as it is currently designed? 
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3. Considering what is both effective and not effective, what can be improved in this 
program? 

4. Considering what is both effective and not effective, what should be improved in 
this program? 

5. How do perceptions of the various stakeholders match up with each other? 

Based upon the questions, the conversations, with the district adnamistrator, the 

evaluators’ notes, and the categorized data from the focus group, the evaluators constructed five 
parallel questionnaires — one for each role group in the program. The initial survey was presented 
to the initial focus group for review and reaction. A panel of experts then reviewed these 
questiormaires, and their feedback resulted in a revision that was submitted to a second panel for 
further critiquing. 

After a second revision, the questiormaires were sent to all participants, approximately 
200 people, for anonymous response. The five separate questiormaires were color coded 
according to the target population intended for the survey — consulting teacher/mentor, 
mentee/newly hired teacher, DRB member, field administrator, and central office administrator. 
The questiormaires were sent with a cover letter encouraging their completion and anonymous 
return to the district administrator. As the completed questiormaires came in, the district 
administrator had them bundled for delivery to the evaluators! Approximately one-half of the 
questiormaires were returned. 

While the responses were being entered into a database, the evaluators reviewed and 
reconfirmed with the initiating district administrator, that the questiormaires were providing the 
type of feedback they had hoped for. The quantitative responses were analyzed through 
descriptive statistics, correlations, and analyses of variance. The quahtative (open-ended) 
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responses were analyzed by initial coding, focused coding, categorization, and having at least 
two, sometimes three, experts in the content area interpret the results. 

The combination of these two approaches is sometimes referred to as mixed methods 
(Newman & Benz, 1998; Newman, Ridenour, Newman, & DiMarco, in press). The mixed 
methods approach, which was applied in this research, generally starts off with a qualitative 
inquuy that leads to a quantitative assessment, and sometimes returns to a qualitative 
interpretation. It is highly heuristic in nature. In this study we started with the qualitative 
question, “What should be evaluated?” which was followed by a quantitative assessment of the 
responses to the questionnaires. These results along with the analysis of the open-ended 
questions resulted in a qualitative-quantitative analysis of the mentoring-evaluation program. 

In keeping with the Personnel Involvement Evaluation Model, the evaluators engaged the 
district administrator to help them interpret the meaning of and to look for “patterns and 
surprises” (Landwehr & Watkins, 1986, preface) in the raw data as it was displayed in frequency 
counts, means, and correlation tables. This allowed the administrator to become more reflective 
about the questions he was interested in and these questions became more focused than the 
previously asked questions. After identifying the statistically significant items in the data 
analyses, the adnimistrator identified 23 specific questions that were to be explored in more 
detail. These questions are provided below along with the responses made by the evaluators. 

The recursive nature of this process, that is, looking at the outcome of the analyses and 
noting the patterns of expected and unexpected results, allowed the adniiiiistrator and the 
evaluators to become more reflective. This is similar to Patton’s (1990) concept of theoretical 
sampling. The administrator had so much information — approximately 100 pages of printouts — 
that he could not necessarily answer the original questions. His knowledge of the participants. 
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however, allowed him to see beyond the data, inferring depth and meaning from it, and allowed 
the evaluators to understand the meaning of the data in ways that could not have existed without 
his participation and that of other members of the focus group. 

Data Sources 

The data sources were the participants separated by roles: interns (newly hired teachers), 
consulting teachers (mentors), and administrators/managers (building administration, central 
office administrations, and members of the board managing the program). 

Findings 

In analyzing the data, the three management groups (field administrators, central officer 
administrators, and members of the DRB) were grouped together because their numbers were so 
few separately that any analytical results would be too highly questionable. 

The direct responses to the 23 specific questions poised during the review of the raw data 
are provided in summary form here, that is, the statistics, charts, and all but one table upon which 
they are based are not provided herein. The extended responses and underl)dng statistics, charts, 
and tables can be obtained from the lead author who may be contacted at ltrenta@uakron.edu. 

1. Which professional development area was most needed? The consulting teachers 
rated the professional development needs of interns more strongly in all four areas — 
planning, delivery, discipline and management, and instructional evaluation — ^than 
did the interns and the administrators. However, the only difference in opinion that 
was statistically significant was that between the consulting teachers and the 
administrators concerning the need for professional development with regard to 
discipline and management. The consulting teachers rated the interns’ need in this 
area significantly stronger than did the administrators. 
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2. 



Which [professional development] need most needs to be met? In general, when 
asked to rate how well professional development needs were met, the interns gave 
the highest ratings and the administrators gave the lowest ratings in all areas. There 
were statistically significant differences between the interns and the administrators in 
the ratings of the interns’ professional development needs being met in Planning and 
Delivery with the interns giving significantly higher ratings in both areas. According 
to the interns and the administrators the professional development need that most 
needs to be met is Planning while the consulting teachers cited Delivery 

3. Are consulting teachers regarded as exemplary teachers? There are no statistically 
significant differences between the groups in their responses. All three groups 
regarded the consulting teachers as exemplary teachers 

4. Were there any differences between respondent groups in the rankings of 
instructional components? The interns ranked feedback on Planning significantly 
lower in importance than did the consulting teachers and the administrators. The 



Table 4b: Rankings of Importance of Feedback 
for Instructional Components 





CTs 


Interns 


Admin. 


Most 

Important 


Planning" 


Discipline & 
Management 


Planning" 




Discipline & 
Management 


Delivery 


Delivery 




Delivery 


Planning" 


Discipline & 
Management® 


Least 

Important 


Instructional 

Evaluation 


Instructional 

Evaluation 


Instructional 

Evaluation 



" Significant difference between interns and both 
consulting teachers and administrators. 

® Significant difference between administrators and both 
consulting teachers and interns. 
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administrators ranked feedback on Discipline and Management significantly lower 
than did the interns and the consulting teachers. 

5. Do the consulting teachers and interns differ in their responses to sufficiency and 
appropriateness of teaching function issues? The interns were significantly less 
certain of what is expected of them in the Teacher Evaluation Program (TEP) than 
were the consulting teachers and the administrators of their roles. There were also 
sigmficant differences in the perceptions about the sufficiency of time in that the 
consulting teachers expressed a significantly stronger disapproval of the statement 
that there is too much time provided for mentoring than do the mtems. The 
administrators agreed with the consulting teachers and significantly differed with the 
interns about the amount of time. 

In the examination of issues related to time there are the next four questions to consider: 

6. Did the interns feel there was enough time for the consulting teachers to observe and 
conference? The consulting teachers did not judge their released time for observing 
and conferencing as sufficient. In this they differed with both the interns and the 
administrators to a statistically significant degree. 

7. Do the interns feel they have enough time to meet the requirements of the TEP? 

8. Do the consulting teachers feel they have enough time to meet the requirements of 
the TEP? 

9. Do the building administrators feel they have enough time to meet the requirements 
of the TEP? 

The consulting teachers, interns, and administrators did judge the time devoted to 
the TEP as sufficient. At the same time, all three groups also agreed, without 
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significant difference, that the TEP required a greater time commitment than they 
had thought it would. 

In terms of the time provided for mentoring, all disagreed with the statement that 
too much time was provided, and the interns and administrators agreed with the 
statement that not enough time was provided. The responses of the interns was 
significantly more negative than those of both the consulting teachers and the 
administrators both as to too much t im e being provided and the need for more time 
for mentoring. 

10. How well did the consulting teachers meet the needs of their interns? Are any areas 
unique? The answer appears to be “quite well.” All three groups agree with that. 
Although the ratings of the administrators are lower than that of the interns and 
consulting teachers in five instances, they are still on the positive side in every 
instance. Dealing with judgments about the quality of the feedback, there are no 
significant differences between the judgments of the three groups in this area. 
Overall the feedback between consulting teachers and interns is judged to be 
excellent for the most part, and it meets the needs of the interns at least 80% of the 
time. 

11. Is there a correlation between the size of the experience difference between the 
interns and the consulting teachers and their ability to positively interact with each 
other? The anonymity of the questionnaires prevented making this comparison. 

12. Entering into the specific area of Program Design Issues, the first question is How 
effective is the information provided to interns regarding the TEP? The consulting 
teachers gave the highest ratings of the three groups to the four survey statements 
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related to the TEP program design. From their perspective, from strongest to weakest 
ratings, they understand what is expected of them, find the handbook helpful, believe 
the process is clearly explained, and receive information on how to prepare for the 
TEP. The administrators’ ratings track those of the consulting teachers but at non- 
significantly lower levels. The interns also gave positive ratings to the four items, 
but all their responses were significantly lower than those both of the consulting 
teachers and of the administrators. Their highest rating was for understanding what 
was expected of them, followed by the process being clearly explained and the TEP 
Evaluation Handbook as helpful. Their lowest rating went to clear inf ormation being 
given on how to prepare for the TEP. Insofar as the interns are concerned the 
information provided to them is effective, but they rate it at a significantly lower 
level of confidence than either the consulting teachers or the administrators. 

13. Does the feedback provided to the interns meet their needs? The interns rated the 
feedback they received as excellent 78.89% of the time and inadequate only 14.87% 
of the time. All groups agreed that the feedback to interns was sufficient to meet 
their needs. 

14. Does the feedback provided to the consulting teachers by the District Review Board 
(DRB) meet their needs? The consulting teachers rated the feedback from the DRB 
as excellent 43.1% of the time and inadequate 25.5% of the time. Across the board, 
the DRB feedback provided to the consulting teachers was judged “excellent” less 
often and “inadequate” more often than any other feedback. 

15. Do the consulting teachers and the administrators agree in their opinions of the 
interns? The consulting teachers judged that their perceptions agreed with those of 
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the administrators 88.49% of the time while the administrators said their perception 
of the interns agreed with that of the consulting teachers 83.1% of the time. The 
difference between the two groups was not significant. 

16. How comfortable are all groups in the combined mentor-evaluator role that 
consulting teachers are fulfilling? The differences in the responses of the three 
groups were not statistically significant. They are more than slightly comfortable 
with the consulting teachers roles as mentor, evaluator, and in the combined role. 
While the difference is not significant, it is interesting to note that it was the 
administrators who expressed the least comfort in the consulting teachers in each of 
the roles and the consulting teachers who expressed the greatest comfort. All three 
groups also express the greatest comfort with the consulting teachers in the role of 
mentor, followed by the combination of roles with the least level of comfort 
expressed for the consulting teachers in the role of evaluator. 

17. How satisfied were the interns regarding the professional development activities 
provided for them? The interns expressed slight to moderate satisfaction with the 
professional development opportunities afforded to them. 

18. How satisfied were the consulting teachers regarding the professional development 
activities provided for them? Consulting teachers more than slightly agree that their 
professional development opportunities have been sufficient and appropriate. Yet, 
they do believe that they would benefit from training other than Pathwise training 
and that they need additional training to be more effective with new teachers. Of the 
possibilities explicitly addressed, the weakest positive agreements for this aspect 
came in response to statements that the coaching training skills was sufficient, that 
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the consulting teachers share strategies, and that the Pathwise tr ainin g was sufficient. 
The interns gave a significantly lower, yet still positive, response to the statement 
that there is a need for additional training for consulting teachers to work effectively 
with new teachers. Administrators rated the statement comparably to the consulting 
teachers. 

Moving to the six general questions that led off the fist of questions from the 
administration, we have these questions. 

19. Were there significant differences of perceptions among surveyed groups? Yes, there 
were. However, the significant differences were in the strength of the mean 
responses to particular questions or statements and not different in the sense of being 
opposite responses except in one case. In that one case the interns slightly disagreed 
with the statement that there was not enough time provided for mentoring while the 
consulting teachers and administrators agreed with the statement. Other than that one 
situation, the differences were between slight agreement and moderate to strong 
agreement with the statements rather then one group agreeing and another group 
disagreeing. 

20. Were there significant differences within groups by assignment, number of years of 
experience, or teaching level? Among the interns, there were no significant 
correlations based on years of experience. There were a number of significant 
correlations between the teaching level of the intern and the responses to some of the 
questions/statements. In general, the significant correlations with the elementary 
level were positive and those with the middle and high school levels were negative 
(that is, the responses tended to be lower than the responses given by those not in the 
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that group). From the middle and high school level responses that had a significant 
correlation, one might generalize that the interns in those levels were more likely 
than the elementary interns to perceived that consulting teachers needed to 
collaborate more and receive more training for working with new teachers. However, 
it must be emphasized again, that the actual responses fiom these groups tended to 
be positive or supportive of the program as it was put into operation. The middle and 
high school level interns simply were not as positive in their expressed perceptions 
as were the elementary level interns. 

Among the consulting teachers, there were no significant correlations based on 
years of experience. There were several significant correlations between the teaching 
level of the consulting teacher and the responses to some of the questions/statements. 
Among the consulting teachers, most of the statistically significant correlations 
between level and responses to questionnaire prompts dealt with feedback, either 
that received by the consulting teachers or their perceptions of the importance of 
feedback related to one of the four teaching functions identified in the TEP. The one 
correlation that did not relate to feedback was that the elementary level consulting 
teachers had a negative correlation with the statement that the CTs share strategies to 
facilitate interns. Between the middle and high school level interns and the 
elementary level consulting teachers, each of the three levels had a group giving 
lower rating to the success of the consulting teachers’ sharing of strategies. 

21 . What information do we have regarding the mentoring-evaluating issue in our 
present format? and 

22. If there were not differences, then the similarities of perceptions should be noted. 
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Probably the most telling data related to these questions were the responses of 
the three groups about their comfort with the consulting teachers role as evaluator 
and the combined role of mentor and evaluator. All groups expressed shght to 
moderate comfort with the roles. The consulting teachers gave the highest rating and 
the administrators the lowest with the interns in-between. None signaled discomfort 
with the roles and the comfort level was well above the lowest comfort level for all 
three groups. All groups were basically comfortable with the combined roles. 

23. How do the comments support the survey responses? The responses to the open- 
ended questions provided support for the survey findings and some hints on ways to 
improve the mentoring-evaluating situation. The recommendations for improving the 
mentoring and evaluating components of the program can be summed up as more 
time, more specifically scheduled time for all stakeholders, more opportunities to 
interact, less paperwork, greater flexibility in dealing with newly hired teachers who 
are experienced and have demonstrated their competence elsewhere. 

Results and Conclusions 

The results fi-om all stakeholders supported the mentor-evaluation program. Some 
stakeholders were more positive about some aspects of the programs than others, but everyone 
believed the program worked. At the same time the perceptions of almost all aspects of the 
program indicated room for improvement. Whether the scale was Strongly Agree to Strongly 
Disagree, Comfortable to Uncomfortable, Sufficient or Appropriate to Insufficient or 
Inappropriate, or Met or Unmet (professional development needs), the mean responses rarely 
were close to the strongest or most positive response. The identification of differences between 
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groups led to interesting discussions between the stakeholders and a better understanding of the 
difference in perceptions. 

One interesting result was the comfort levels of all stakeholders in having the mentors 
serve in dual roles, as a mentor and as an evaluator of his or her intern (new teacher). Mentors’ 
successfully serving as evaluators is contrary to most of the current research about mentoring. In 
this case of added note is that it was the administrators who were least comfortable with 
consulting teachers being both mentors and evaluators, although their comfort level was not 
significantly different than either the interns or the consulting teachers. 

Another issue that became evident was the lack of flexibility caused by requiring the 
application of the TEP process to all new teachers in the same way, that is, by not allowing for 
difference in treatment between a newly hired teacher who is a first year teacher and a newly 
hired teacher who was a successful, experienced teacher in another district. 

Two anticipated results were that time is an issue, as is the amount of record keeping. 
There were three interrelated issues raised about time: (a) the need to prioritize time for the 
mentoring process, (b) the need to provide more time for the consulting teachers to devote to the 
program, and (c) the need for consulting teachers and administrators to prioritize time to deal 
with the evaluation requirements. 

The record keeping paperwork was considered too time consuming and not relevant to 
improving performance. 

Use of Evaluation Results 

Again, in line with the Personnel Involvement Evaluation Model, the final report 
prepared by the evaluators was delivered to and discussed with the district administrator. The 
executive summary, which included detailed responses to the 24 specific questions as well as the 



original five focusing questions, was provided to the DRB. The lead evaluators met with the 
DRB and discussed the findings and recommendations with them. The members recognized the 
overall positive nature of the responses from all participant groups. They also noted that the 
changes for improvement of the program illuminated by the results of the evaluation were subtle 
rather than substantial. The DRB intended to use the executive summary both to stimulate 
conversations with the larger stakeholder group and to report the state of the TEP program to 
their board of education. 

Limitations of the Evaluation 

The most significant limitation was that the evaluation of program effectiveness was tied 
to the perceptions of the stakeholders rather than to measures of behavior, due to a short time 
firame for the evaluation and limited funds. The effort to overcome this limitation included 
aggregating the scores (rankings and ratings) and providing for reliability and validity checks 
using a focus group, expert judges, and piloting the instruments. 

Another limitation is the probable presence of aspects of the Hawthorne Effect. That is, 
some participants can be expected to respond to the surveys in what they perceived to be the 
socially desirable manner rather than expressing their own judgments. On the other hand, people 
who have hostile feelings toward the district or the district management can be expected to 
respond in ways that would be perceived as denigrating the program. They express their anger 
rather than their judgment. The same efforts used to overcome the use of stakeholder perceptions 
rather than hard measures of behavior are useful in overcoming the desire to give the socially 
acceptable response and the desire to express anger or inflict harm rather than to express 
considered judgment. 
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To give fullest assurance of anonymity, the siurvey forms did not have a place for name or 
signature and the forms and return envelopes were not coded in any way. This meant that it was 
not possible to directly encourage those who did not return the survey to fill it out. Also, it was 
not possible to solicit opinions from those who did not return the survey by another means, such 
as a telephone survey. As a result, the evaluation is based on the perceptions of those who chose 
to respond to the survey and does not consider the opinions of those who chose not to respond. 

Due to the small number of administrators in the district, in analyzing the data it was 
necessary to combine the administrator groups (central office and building) in order to have a 
sufficient number for the statistical analysis to have a reasonable degree of certainty. 

Significance 

There is educational significance to this study on two levels. First, it is an example, a case 
study, of using a highly participatory process for the formative evaluation of a teacher induction 
program. The education of the evaluators concerning the program enabled them to more 
effectively meet the needs of the stakeholders/client. The step-by-step interaction between the 
evaluators and the client administrator and the stakeholder focus group enabled stakeholders’ 
questions about the program to become more targeted and more insightful. Mixing qualitative 
data collection and analysis with quantitative methods both longitudinally in the course of the 
evaluation and latitudinally in the questionnaires provided a final product that was highly 
meaningful and useful to the stakeholders. Particularly encouraging was that the meanings or 
concepts held by individuals appeared to broaden as the stakeholders listened to and embraced 
the meanings of others in the course of this evaluative process. 

Second, the finding that all groups were comfortable with the mentor also being a 
participant in the evaluation of the interns contradicts the conventional wisdom that the roles 
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should be separated. Particularly interesting is that the two groups expressing the highest level of 

comfort with the mentor-evaluator combined role were those most intimately involved in it, the 

interns and the consulting teachers. 
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